class: title-slide <br> <br> # Effects of Early Warning Emails on Student Performance <br> .padding_left.pull-down.white[ .bold[_J. Klenke_], T. Massing, N. Reckmann, J. Langerbein, B. Otto, M. Goedicke, C. Hanck <br> <br> <br> `\(15^{TH}\)` International Conference on Computer Supported Education Prague, 21-23 April, 2023 ] --- # Research Idea and Course Description - __Research Idea:__ students should receive objective and motivating warning email - Analyzed Course: _Inferential Statistics_ at the University of Duisburg-Essen of the summer semester 2019 - Compulsory course for several business and economics programs - Weekly 2-hour lecture - Weekly 2-hour exercise - [Kahoot!](https://kahoot.com/) games were used to interact with students during classes - Homework and 5 online tests were offered on the e-assessment platform [JACK](https://s3.paluno.uni-due.de/en/forschung/spalte1/e-learning-und-e-assessment) - Information from __802__ individuals were collected from Moodle - __337__ students took an exam at the end of the semester --- # Data and Decision Rule - We used two data sources - First three online test results - Cumulative points of the tasks in JACK - Logit model was used to predict students probability to pass the final exam based on the frist 3 online tests - Model was trained with the latest data obtained from previous edition of the same course - If predicted probability to pass `\(\leq 0.4\)` the student got a warning mail --- # Course Timeline Main Events <br> <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#plots/timeline_plot.png" alt="Timeline for the key events in the 2019 summer term course Inferential Statistics (treatment cohort)." width="80%" /> <p class="caption">Timeline for the key events in the 2019 summer term course Inferential Statistics (treatment cohort).</p> </div> - The shaded area indicates the period after treatment. - 57 days between the warning mail and 1st exam - 113 days between the warning email and 2nd exam --- # Literatur - Purdue -> Problem : Kaum analisiert --- # RDD Example Plot — I ## Parametric Estimation <img src="data:image/png;base64,#plots/late_tikz1.png" width="80%" style="display: block; margin: auto;" /> .pull-right[ - Groups left and right of the cutoff - Local Average Treatment Effect (__LATE__) ] --- # RDD Example Plot — II ## Non-parametric Estimation <img src="data:image/png;base64,#plots/non_p_late_tikz1.png" width="80%" style="display: block; margin: auto;" /> -- - We employed the data-driven approach by <a id='cite-imbensoptimal2009'></a><a href='#bib-imbensoptimal2009'>Imbens and Kalyanaraman (2009)</a> to determine the bandwidth ??? - The method fits the bandwidth as widely as possible without introducing other confounding effects --- # Model Assumptions - The running variable `\(W\)` (predicted probability to pass the exam) needs to be continuous around the cutoff, otherwise students could manipulate the treatment .pull-left-2[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#plots/test_cont_label.png" alt="Graphical illustration of the McCrary sorting test." width="80%" /> <p class="caption">Graphical illustration of the McCrary sorting test.</p> </div> ] -- .pull-right-1[ <br> - There is no jump in the density around the cutoff point of `\(0.4\)`. - `\(p\)`-value: `\(0.509\)` - The incentive to manipulate the treatment is quiet low. ] -- .pull-down[ - Also standard IV estimation assumptions must hold, which they do by construction of RDD ] --- # Empirical Results .pull-left-2[ <br> <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#plots/model_plot_label.png" alt="Graphical illustration of the RDD model." width="80%" /> <p class="caption">Graphical illustration of the RDD model.</p> </div> ] -- .pull-right-1[ ### Estimates - LATE: 0.193 (4.889) - Bandwidth: 0.255 - `\(F\)`-statics: 0.257 - `\(N\)` : 126 <br> - We also estimated the RDD with covariates - The results were identical ] ??? - theoretisch sollten Sie jetzt einen großen Sprung sehen --- # Empirical Results - The LATE estimate is positive but not significant - An estimate of `\(0.193\)` means that students who received the warning email achieved `\(0.193\)` points more than comparable students who did not - Compared to the `\(60\)`-point exam, the effect size seems limited - The bandwidth of `\(0.255\)` was determined with the data-driven approach of <a href='#bib-imbensoptimal2009'>Imbens and Kalyanaraman (2009)</a>. - Only students with a predicted probability `\(0.4\)` (cutoff) `\(\pm 0.255\)` (bandwidth), are included in the analysis. - This leads to the sample size of `\(126\)` students. --- # Discussion - Our RDD results do not provide evidence that the warning email has a significant effect on students’ results (or behavior) - The variance around the cutoff is rather high, which compromises the detection of an effect - Many individuals are not included in the final analysis for several reasons - Students dropping the course - Students far away from the cutoff are not provding much information for the model -> Thus precise estimation of the treatment becomes more difficult --- # Discussion - Students get also feedback through their online tests - The warning might may also lead weak students to postpone participation to a later semester - The motivation and objective feedback from one warning email is rather small --- # Further Research - The effect on the dropout rate from such warning emails or systems requires further attention - An automatic repeatedly feedback system could possibly have a greater impact on students motivation - A detialed recurring feedback could also used to guide students -- <br> .blockquote[ We see the open and transparent communication of the student’s performance to the students as a positive aspect of the system. ] --- # Sources .font90[ - Angrist, J. D., Imbens, G., and Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434):444–455. Publisher: Taylor & Francis. - Huntington-Klein, N. (2022). The effect : an introduction to research design and causality. Chapman & Hall book. CRC Press, Taylor & Francis Group, Boca Raton ; London ; New York, first edition edition. - Imbens, G. and Kalyanaraman, K. (2009). Optimal bandwidth choice for the regression discontinuity estimator. National Bureau of Economic Research, 1(14726). - McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2):698–714. ] --- # References Imbens, G. and K. Kalyanaraman (2009). "Optimal Bandwidth Choice for the Regression Discontinuity Estimator". In: _National Bureau of Economic Research_ 1.14726. --- # Appendix --- # Regression Discontinuity Design (RDD) - The treatment is __not__ randomly assigned and therefore methods like OLS are not suitable - Treatment is a function of the predicted probability to pass the exam - Consider the following _sharp_ RDD representation <a id='cite-Huntington-Klein2022'></a><a href='#bib-Huntington-Klein2022'>Huntington-Klein (2022)</a>: `$$Y_i = \beta_0 + \alpha T_i + \beta W_i + u_i$$` .padding_left.padding_left.font80[ <ul> <li> \(W_i\) denotes the predicted probability to pass the final exam</li> <li> \(T_i\) indicates if a student received a mail</li> <ul style="list-style-type: '‣ ';" > <li>\(T_i = 1[W_i \leq c]\) , with \(c = 0.4\) </li> </ul> <li>\(\alpha\) denotes the treatment effect</li> <li>\(u_i\) denotes the error term</li> </ul> ] -- <svg viewBox="0 0 192 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#004c93;" xmlns="http://www.w3.org/2000/svg"> <path d="M176 432c0 44.112-35.888 80-80 80s-80-35.888-80-80 35.888-80 80-80 80 35.888 80 80zM25.26 25.199l13.6 272C39.499 309.972 50.041 320 62.83 320h66.34c12.789 0 23.331-10.028 23.97-22.801l13.6-272C167.425 11.49 156.496 0 142.77 0H49.23C35.504 0 24.575 11.49 25.26 25.199z"></path></svg> `\(\;\)` This design is not suitable for our analysis as our groups are not perfectly separated --- # Fuzzy RDD - __Fuzzy__ RDD allows to analyse a treatment in a setting where the two groups are not perfectly separated - Only the likelihood of receiving the treatment needs to _change_ - The effect is estimated through an instrumental variable estimation <a id='cite-angristidentification1996'></a><a href='#bib-angristidentification1996'>Angrist, Imbens, and Rubin (1996)</a> where in the first stage the `\(\widehat{T}_i\)` are estimated which then are inserted in the second stage - First Stage: `$$T_i = \gamma_0 +\gamma_i Z_i + \gamma_2 W_i + \nu_i \qquad \quad$$` - Second Stage: `$$Y_i = \beta_0 + \alpha \widehat{T}_i + \delta_1 W_i + \beta X_i + u_i$$` --- # Regression Discontinuity Design (RDD) II .font70[ <ul> <li>RDD compares the individuals around the cutoff to estimate the effect</li> <li><strong>Main Assumption:</strong> Individuals around the cutoff are compareable and only differ in the treatment assignment</li> <ul> <li>The estimate is called Local Average Treatment Effect (<strong>LATE</strong>) </li> </ul> <li>Two estimation approaches: parametric and non-parametric</li> <ul> <li>Parametric estimation</li> <ul> <li>Uses the whole sample size but (many) more parameters</li> <li>Individuals away from the cutoff are less relevant for the estimation of the effect</li> </ul> <li>Non-parametric estimation</li> <ul> <li>Only <em>comparable</em> (near the cutoff) individuals are used for the analysis</li> <ul> <li>Decision whether to include an individual depends on the running variable \(W\)</li> <li>The groups are determined by the the data-driven approach of Imbens and Kalyanaraman (2009)</li> <ul> <li>With an \(F\)-test the bandwidth is determined</li> <li>\(c \pm bandwidth\) are the two groups</li> </ul> </ul> </ul> </ul> </ul> ]